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We predict a new class of three dimensional topological insulators in thallium-based in-V-Vl2 
ternary chalcogenides, including TlBiQ2 and TlSbQ2 (Q = Te, Se and S). These topological insu- 
lators have robust and simple surface states consisting of a single Dirac cone at the F point. The 
mechanism for topological insulating behavior is elucidated using both first principle calculations 
and effective field theory models. Remarkably, one topological insulator in this class, TlBiTe2 is also 
a superconductor when doped with p-type carriers. We discuss the possibility that this material 
could be a topological superconductor. Another material TlSbS2 is on the border between topolog- 
ical insulator and trivial insulator phases, in which a topological phase transition can be driven by 
pressure. 

PACS numbers: 



Topological insulators have attracted great attention 
in condensed matter physicsi. Since the first the- 
oretical prediction^ and the subsequent experimental 
observation^ in HgTe quantum wells, several other topo- 
logical insulators in three dimensional (3D) bulk materi- 
als have been theoretically predicted and experimentally 
observed^—. In particular, tetradymite semiconductors 
Bi2Te3, Bi2Se3, and Sb2Te3 are predicted to be topolog- 
ical insulators with a large bulk band gap whose surface 
state consists of a single Dirac cone^. The mechanism 
for the topological insulating behavior in these 3D mate- 
rials is the band inversion at the F point caused by large 
spin-orbit coupling, similar to the mechanism first dis- 
covered in HgTe quantum well&2,. The tetradymite semi- 
conductors have a layered structure consisting of stacking 
quintuple layers, making surface preparation particularly 
simple. 

In this work we predict a new class of 3D topolog- 
ical insulators in the thallium-based III-V-VI2 ternary 
chalcogenides. These inversion symmetric topological in- 
sulators have a bulk energy gap and topologically pro- 
tected surface states consisting of a single Dirac cone. 
Unlike the tetradymite semiconductors, these materials 
are intrinsically 3D, and do not have a weakly coupled 
layer structure. Nonetheless, effective field theory model 
describing the band electrons close to the Fermi energy 
takes the same form as the model proposed earlier for 
the tetradymite semiconductors^, and the mechanism for 
topological insulating behavior can be understood in a 
similar way. 

The discovery of topological insulators also inspired the 
intense search for topological superconductors^rJ^. Time 
reversal invariant topological superconductors have a full 



pairing gap in the bulk and topologically protected sur- 
face states consisting of Majorana fermions, see Fig 1 
of Ref.^. Wheareas Dirac fermions have both particle 
and hole types, Majorana fermions are their own anti- 
particles.^^ In the simplest version, the surface state of 
a 3D topological superconductor consists of a single Ma- 
jorana cone, thus containing half the degree of freedom 
of the Dirac surface states of a simple 3D topological 
insulator. This fractionalization of the degree of free- 
dom introduces quantum non-locality and is central to 
the program of topological quantum computing based on 
Majorana fermions^^. Superfluid He3B phase has been 
proposed as an candidate for the 3D topological super- 
fluid state, however, no example of a topological super- 
conductor state has been found so far. 

The most striking property of this new class of topo- 
logical insulators is the superconductivity observed in p 
doped TlBiTesr^. In this paper, we propose that the p 
doped TlBiTe2 could inherit the topological properties 
from its parent topological insulator, and outline pair- 
ing scenarios under which the topological superconductor 
state could be realized. 

Thallium-based III-V-VI2 ternary chalcogenides have 
rhombohedral crystal structure with the space group D^^ 
(R3m), which is similar to tetradymite semiconductors. 
We take TlBiTe2 as an example. The crystal structure of 
TlBiTe2 can be viewed as the distorted NaCl structure 
with four atoms in the primitive unitcell, which is shown 
in Fig 1 (a). These atoms are placed in layers normal 
to the three-fold axis with the sequence -Tl-Te-Bi-Te-— . 
For Bi2Se3, five atomic layers form a quintuple layer and 
the coupling between two quintuple layers is very weak, 
of the van der Waals type^. However here the outmost 
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FIG. 1: Crystal structure, (a) Crystal structure of TlBiTe2 
with three primitive lattice vectors denoted as al;2;3, in- 
cluding four atoms, (b) The equivalent hexagonal lattice of 
TlBiTe2 with atomic layers stacked in the sequence -Tl-Te- 
Bi-Te-. 



electron shells for Tl, Bi and Te are all p-orbitals (6p^ 
for Tl, for Bi and for Te) and each Tl (Bi) layer 
is sandwiched by two Te layers, therefore there is strong 
coupling between every two atomic layers for TlBiTe2 
and the crystal structure is essentially 3D. Figure 1(b) 
shows the 3D layered structure with the rhombohedral 
unit cell of TlBiTe2. Similar to Bi2Se3, there is inversion 
symmetry for this type of materials and both Tl and 
Bi sites act as the inversion center under an inversion 
operation. 

To investigate the electronic band structure and topo- 
logical property of these materials, we performed ab ini- 
tio calculations within the density functional theory us- 
ing the Perdew- Burke-Ernzerhof type generalized gradi- 
ent approximation^^and the projected augmented wave 
method^^ implemented in the Vienna ab initio simula- 
tion package^^ . The plane wave basis is used with en- 
ergy cutoff of 300 eV. Spin-orbit coupling (SOC) is in- 
cluded except in ionic optimization. We optimized of lat- 
tice parameters and ionic positions first, and then used 
the relaxed structure to calculate electronic properties. 
The optimized lattice parameters agree with reported 
experimental^^^^^^^^ and theoretical^^ results. Our cal- 
culated band structures for these six materials are also 
consistent with previous calculations^^. The electronic 
band structures of TlBiSe2 and TlBiTe2 without (dashed 
line) and with SOC (solid line) are shown in Fig. 2 (a) 
and (b), respectively. There is a large energy shift for the 
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FIG. 2: Band structure, surface state and Brillouin zone. 
Band structure for (a) TlBiSe2 and (b) TlBiTe2 without (gray 
dotted lines) and with (red solid lines) SOC. The dashed line 
indicates the Fermi level, (c) The surface states (red line) 
shows a single Dirac cone for TlBiSe2. The black line shows 
the band edge for conduction band and valence band. The 
dispersion of surface states are calculated from the effective 
Hamiltonian (U}. (d) Brillouin zone for this class of ma- 
terials. The four inequivalent time- reversal- invariant points 
are r(0;0;0), L(7r; 0; 0), F(7r; tt; 0) and T(7r;7r;7r). We further 
project the 3D bulk Brillouin zone into the plane of the atomic 
layer to obtain the surface Brillioun zone. 



conduction and valence band and the band gap shrinks 
considerably when the SOC is turned on, indicating that 
the SOC plays an important role for this type of mate- 
rials. The conduction band minimum (CBM) lies at F 
point for both TlBiSe2 and TlBiTe2, while the valence 
band maximum (VBM) is at F point for TlBiSe2 and 
along the L-T line for TlBiTe2. Therefore TlBiSe2 is a 
direct gap semiconductor while TlBiTe2 is an indirect 
narrow gap semiconductor, consistent with previous ex- 
periments and calculations^^^^. In order to investigate 
the topological property of the system, we analyze the 
charge density of the VBM and CBM at the F point. 
Before SOC is included, the VBM is composed mainly 
by the Te-p orbital and CBM mainly by the Bi-p and 
Te-sp orbitals. After SOC is turned on, VBM and CBM 
exchange their charge density characteristics, indicating 
that a band inversion may occur, just as in the case of 
HgTe quantum wells and Bi2Se3 type of materials. To 
further determine the topological nature of the system, 
we follow the parity criteria proposed by Fu and Kane^, 
and calculate the product of the parity eigenvalues of 
the Bloch wavefunction for the occupied bands at all 
time-reversal-invariant momenta F, L, F, T in the Bril- 
louin zone, both with and without SOC. For TlSbS2, the 
product of the parity eigenvalues remains the same when 
the SOC is included, while for all the other materials 
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FIG. 3: Table: the parity of the band at the F point and 
the band gap for the six materials. Here, we show the parity 
eigenvalues of ten occupied bands and the lowest unoccupied 
band. The product of the parity eigenvalues for the ten oc- 
cupied bands is given in brackets on the right of each row. 
We list the band gap both at the F point and in the whole 
Brillouin zone (bulk gap) in unit of eV. 



in this class, the parity eigenvalue of one occupied band 
(VBM) at F point changes after turning on SOC, and the 
parity eigenvalues for all the occupied bands at L, F, T 
do not change. The parity eigenvalues of the total 10 
valence bands and the first conduction band at F point, 
as well as the band gaps are listed in Table 1. From the 
parity analysis we find that TlSbS2 is a trivial insulator 
and all the other materials in this class are strong topo- 
logical insulators. Although TlSbS2 is a trivial insulator, 
it sensitively depends on the lattice constant. We find 
that about 2% compressive strain along the c-axis in the 
hexagonal lattice will induce a band inversion between 
the conduction and valence bands and the correspond- 
ing pressure is less than 2 GPa, which is reachable under 
ambient condition. Therefore this material opens the ex- 
citing possibility to systematically study the topological 
quantum phase transition between 3D topological insu- 
lators and normal insulators by tuning pressure. 

As discussed above, the band inversion only occurs at 
the F point, therefore it is helpful to investigate the ef- 
fective Hamiltonian near the F point, which can be con- 
structed from the symmetry property of the system. To 
derive the effective Hamiltonian, we need to identify at F 
point the representations of the crystal symmetry group 
for both the conduction and the valence bands. Here 
we denote the conduction band as P1+ band and the 
valence band as P2~ band, where ± denotes the par- 
ity of the corresponding bands. With the spin degener- 
acy, four bands ((|Pl+,t) , \P2-,t) , 1^^1+4} , IP'^'A))) 
need to be taken into account for the effective Hamilto- 
nian. The crystal structure of TlBiTe2 belongs to 
which is same to that of Bi2Se3. Thus the wave functions 



at F point can also be classified according to the symme- 
try group of 1^1^, similarly as we have done for Bi2Se3^^. 
According to the irreducible representation of the space 
group D^^^ we find that the conduction band belongs to 
the 
the F, 

tive Hamiltonian of Bi2Se3 is still valid here, which in the 
basis of (|Pl+,t) , |P2-,t) , |P1+,;) , |P2-,;)) reads^i^ 
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representation while the valence band belongs to 
g representation. Consequently, the four band effec- 
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with k± = kx ^ 
M{k) = M - Biki - 132 
Here i^o(k) is the effective Hamiltonian expanded up 
to the k quadratic term, which preserve the in-plane 
rotation symmetry. i^3(k) is the /c-cubic term which 
breaks the in-plane rotation symmetry down to the three 
fold rotation symmetry. Combining the k • p pertur- 
bation theory with the ah initio calculation^^, we can 
numerically calculate the parameters of our model, giv- 
ing C = -OMbeV, M = 0.087ey, Ai = 1.330eV-A, 
A2 = 2.821ey-A, Di = 6.338el/-A2, D2 = ll.UOeV-A'^, 
Bi = 0.342ey•A^ P2 = 18.225eV-A2, = U.367eV- 
and R2 = 43.331eV-A^ for TlBiTe2. The parameters 
M > 0, Pi > and P2 > indicate that the system 
stays in the inverted regime and is topologically non- 
trivial. The topologically non-trivial surface states can 
be directly calculated from the above four band model 
by imposing proper boundary conditior£i21i2^. Taking a 
semi-infinite sample where the Hamiltonian ([!]) applies 
only for z > 0, it can be shown that two localized states 
It/j^) and It/ji) appear dit kx = ky = which are time- 
reversal partner of each other. We can further project 
the effective Hamiltonian ^ onto the subspace spanned 
by these two localized states, which yields^^ 



C^D2kl^A2{kx(Jy 



f(4 



fc3)<j,(4) 



up to k^. Here the parameters (7, P2, ^2, and Pi depend 
on the detail of the boundary condition^^ and material 
parameters'^. For simplicity, we just take bulk values 
for these parameters, the energy dispersion for the topo- 
logically nontrivial surface states are plotted in the Fig 
2(c), which shows a single Dirac cone at F point, simi- 
lar to the case of Bi2Se3. For Bi2Se3, due to the weak 
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FIG. 4: Superconductivity in TlBiTe2. a. Bulk and surface 
states of TlBiTe2. Ev and Er are the valence band top in 
the whole Brillouin zone and at F point, respectively, /ii and 
/i2 labels two typical positions of chemical potential, b and c. 
Fermi surface shape corresponding to /xi and /i2 respectively, 
d. Schematic picture of the He^-B type triplet pairing with 
spin direction correlated with momentum. 

coupling between two quintuple layers, the clean surface 
can be easily obtained in experiment. For TlBiSe2, as 
we have pointed out above, every two atomic layers are 
strongly coupled, and consequently, trivial surface states 
with dangling bonds may exist on a cleaved surface. How- 
ever these trivial surface states will not change the topo- 
logical property of the system. 

Among the predicted materials, TlBiTe2 is observed to 
be a superconductor with p-type carrier. TlBiTe2 has 
an indirect gap, with 6 hole pockets around the T point 
upon hole doping, as shown in Fig. 4 b and c. Com- 
pared with the other five materials in this family, which 
either have an indirect gap comparable with the F point 
gap or have a direct gap, it is natural to expect that 
the 6 hole pockets around the T point are responsible 
for the superconductivity in TlBiTe2. Depending on the 



position of the chemical potential, there are two distinct 
superconducting phases possible, as shown in Fig. 4 a. 
First, when the chemical potential lies below the top of 
the valence band Ey^ but still above the top of the va- 
lence band at the F point Er^ topological surface states 
around the F point coexist with the 6 bulk hole fermi 
pockets. In the superconducting state, the surface states 
can also become superconducting due to proximity effect 
with the bulk states. As proposed by Fu and Kane^^, 
each vortex of such a superconductor has a Majorana zero 
mode, making this system a new candidate for topologi- 
cal quantum computation^^. Compared to the Cu doped 
Bi2Se3 superconductor discovered recentl}^, the coexis- 
tence between topological surface states and bulk super- 
conductivity in TlBiTe2 is much better defined, because 
of the well-separation of the surface and bulk states in 
momentum space. Second, when the chemical potential 
lies below Er^ the topological surface states are not well- 
defined any more, and a bulk hole pocket appears around 
F point. When the hole pocket around F point is small 
compared to the pre-existing 6 hole pockets, the pairing 
symmetry at the F point pocket is determined by that 
of the 6 pockets through proximity effect in momentum 
space. If the 6 hole pockets have the same sign of pairing 
amplitude, the resulting superconducting state is topo- 
logically trivial. However, since Tl d-orbital characters 
are generally present for the wave functions of the hole 
pockets. Coulomb correlation effects maybe important, 
and inter-pocket repulsive scattering generally prefer op- 
posite signs of pairing amplitudes on different pockets. 
This mechanism implies a negative "Josephson coupling" 
between different pockets. However, due to the three- fold 
symmetry of the fermi surface, such a coupling is frus- 
trated, so that the pairing order parameter in the ground 
state may become complex. A natural choice of such a 
complex orbital pairing symmetry without breaking the 
time reversal symmetry is a triplet pairing symmetry sim- 
ilar to the BW state in He3 B-phase, as illustrated in Fig. 
4 d. When the hole pocket around Gamma point appears, 
the proximity effect from such a pairing symmetry leads 
to a topological superconductor with a nodeless bulk gap, 
and a gapless surface state consisting of a single branch 
of Majorana fermions. 
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